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The Impact of a Reading Intervention 
for Low-Literate Adult ESL Learners 



Executive Summary 



According to the 2008 program year statistics from the U.S. Department of 
Education (ED), 44 percent of the 2.4 million students in the federally funded 
adult education program in the United States were English as a second language 
(ESL) students (ED, 2010). Of these, about 185,000 were at the lowest ESL level, 
beginning literacy. These students, many of whom face the dual challenge of 
developing basic literacy skills — including decoding, comprehending, and 
producing print — along with proficiency in English, represent a range of 
nationalities and cultural backgrounds. Although the majority of students come 
from Mexico and other Spanish-speaking countries, there are also students from 
Africa, India, the Philippines, China, Vietnam, and the Caribbean (Wrigley, 

Richer, Martinson, Kubo, & Strawn, 2003). 

Adult basic education (ABE) and ESL programs, authorized by the Workforce 
Investment Act and also funded with state and local funds, are designed to assist 
students in their efforts to acquire literacy and language skills by providing 
instruction through local education agencies, community colleges, and 
community-based organizations. The content of instruction within ESL classes 
varies widely. It is often designed to assist students in their efforts to acquire 
literacy and language skills by providing a combination of oral language, 
competency-based work skills, and literacy instruction (Condelli, Wrigley, Yoon, 
Cronen, & Sebum, 2003). There is, however, little rigorous research that identifies 
effective instruction. A comprehensive review of published research studies on the 
effects of literacy interventions for ABE and adult ESL learners (Condelli & 
Wrigley, 2004) found that out of 17 adult education studies that used a rigorous 
methodology (i.e., quasi-experimental or randomized trials), only 3 included adult 
ESL learners (Diones, Spiegel, & Flugman, 1999; St. Pierre et al., 1995; St. Pierre 
et al., 2003). Furthermore, among the 3 studies that included adult ESL learners, 
only 1 presented outcomes for those learners, and that study experienced 
substantial methodological problems that limited the validity of the findings (e.g., 
a 40 percent overall attrition rate and different attrition rates in the intervention vs. 
control groups; Diones et al., 1999). 

To help improve research-based knowledge of effective instruction for 
low-literate ESL learners, the National Center for Education Evaluation and 
Regional Assistance of ED’s Institute of Education Sciences contracted with the 
American Institutes of Research (AIR) to conduct a Study of the Impact of a 
Reading Intervention for Low-Literate Adult ESL Learners. The intervention 
studied was the basal reader Sam and Pat, Volume I, published by Thomson- 
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Heinle (2006). The study team consisted of AIR, Berkeley Policy Associates 
(BPA), the Lewin Group, Mathematica Policy Research, Inc., Educational Testing 
Service (ETS), and World Education. 

The goal of this study was to test a promising approach to improving the literacy 
skills of low-literate adult ESL students under real-world conditions. In their 
review of the research on ESL instruction in related fields, including adult second 
language acquisition, reading and English as a foreign language instruction, 
Condelli & Wrigley (2004) concluded that instruction based on a systematic 
approach to literacy development was a promising intervention for low-literate 
adult ESL learners that would be valuable to study (Brown et al., 1996; Cheek & 
Lindsay, 1994: Chen & Graves, 1995; Carrell, 1985; Rich & Shepherd, 1993; 
Roberts, Cheek & Mumm, 1994). Specifically, the factors identified as defining a 
systematic approach to literacy development included: (1) a comprehensive 
instructional scope that includes direct instruction in phonics, fluency, vocabulary 
development and reading comprehension, (2) a strategic instruction sequence, 

(3) a consistent instructional format, (4) easy-to-follow lesson plans, and 
(5) strategies for differentiated instruction. 

Sam and Pat was selected as the focus of the study because it offers an approach 
to literacy development that is systematic, direct, sequential, and multi-sensory. It 
also includes multiple opportunities for practice with feedback. Consistent with 
characteristics identified as promising by Condelli & Wrigley (2004), Sam and 
Pat provides opportunities for cooperative learning, real world tasks, and an 
explicit focus on reading. In addition, the text was developed for and had been 
used by the developers with students similar to the study population (literacy level 
ESL learners). 

The impact study used an experimental design to test the effectiveness of Sam and 
Pat in improving the reading and English language skills of adults enrolled in 66 
ESL literacy classes at 10 sites. The study addressed three key research questions: 

1. How effective is instruction based on the Sam and Pat textbook in 
improving the English reading and language skills of low-literate adult 
ESL learners compared to instruction normally provided in adult ESL 
literacy classes? 

2. Is Sam and Pat effective for certain subgroups of students (e.g., native 
Spanish speakers)? 

3. Is there a relationship between the amount of instruction in reading or 
English language skills and reading and English language outcomes? 
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This report describes the implementation of Sam and Pat at the study sites, 
compares the instruction and student attendance in Sam and Pat classes with that 
in the standard adult ESL classes, and examines the impact of Sam and Pat on 
reading and English language outcomes. In addition, the report examines the 
relationship between instruction, attendance, and student outcomes. 

The study produced the following key results: 

❖ More reading instruction was observed in Sam and Pat classes , while 
more English language instruction was observed in control classes. The 
Sam and Pat classrooms spent more time on reading development 
instruction (66 percent of observed intervals in Sam and Pat classrooms 
compared to 19 percent in control classrooms), and the difference was 
statistically significant. Conversely, the control classrooms spent more 
time on English language acquisition instruction (68 percent of observed 
intervals in control classrooms compared to 27 percent in Sam and Pat 
classrooms), and this difference was also statistically significant. 

❖ Although students made gains in reading and English language skills, 
no differences in reading and English language outcomes were found 
between students in the Sam and Pat group and students in the control 
group. On average, students participating in the study made statistically 
significant gains in reading and English language skills over the course of 
the term (effect sizes of 0.23 to 0.40). However, there were no statistically 
significant impacts of Sam and Pat on the reading and English language 
outcomes measured for the overall sample. 

❖ There were no impacts of Sam and Pat on reading and English 
language outcomes for five of six subgroups examined. For students 

with relatively lower levels of literacy at the start of the study, there was 

2 

some suggestive evidence of a positive impact on reading outcomes . 
Among students with lower levels of literacy at the beginning of the term, 
Sam and Pat group students scored higher on the Woodcock Johnson 
word attack (decoding) assessment than control group students (effect 
size = 0.16). Because this difference was not significant after adjusting for 
multiple comparisons, however, it is possible that the effect is due to 
chance alone. 



2 Lower literacy was defined as scoring at a Grade 2 equivalent or below on the Woodcock 
Johnson Letter-Word Identification and Word Attack subtests (raw scores of 31 and 9, 
respectively). 
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Summary of Study Design and Methods 



The study was designed to estimate the impact of Sam and Pat relative to standard 
ESL instruction (i.e., the kind of instruction ESL students in study sites would 
receive in the absence of the study) on reading and English language outcomes. 

The evaluation employed a randomized research design that included the 
following: 

❖ 10 adult education program sites; 

❖ 33 teachers; 

❖ 66 classes; and 

❖ 1,344 low-literate adult ESL learners. 

The program sites were a purposive sample. From among the states with the 
largest adult ESL enrollments, we selected sites that had enrollments of adult ESL 
literacy learners large enough to support the study design, 2 or more classes for 
ESL literacy students that met at the same time and in the same location, and an 
enrollment process that would accommodate random assignment. 

Within each site, teachers and students were randomly assigned to one of 
two groups: 

❖ The Sam and Pat group, which was intended to include a minimum of 
60 hours of Sam and Pat - based instruction per term, with any remaining 
class time being spent on the standard instruction provided by the 
program; and 

❖ The control group, which consisted of the standard instruction provided by 
the program. 

Teachers (or classes) within each program site were randomly assigned in pairs, 
so that, within each pair, the Sam and Pat and control class met at the same time, 
in the same or an adjacent building, and for the same number of hours. Data 
collection for the study occurred between September 2008 and May 2009 with 
two cohorts of students, one that attended in fall 2008 and the second in spring 
2009. Students were tested on the study’s battery of assessments, which included 
tests of reading and English language skills at the beginning of the term and after 
about 12 weeks of instruction. A description and schedule for the study’s data 
collections are provided in Table ES.l. 
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The following tests were selected to measure the range of skills that could 
potentially be impacted by Sam and Pat - based instruction: 

Reading Skills 

❖ Woodcock-Johnson Letter-Word Identification (WJID; Woodcock, 
McGrew, & Mather, 2001) 

❖ Woodcock-Johnson Passage Comprehension (WJPC; Ibid.) 

❖ Woodcock-Johnson Word Attack (WJWA; Ibid.) 

❖ SARA Decoding (SARA Dec; Sabatini & Bruce, in press) 

English Language Skills 

♦> Oral and Written Language Scales (OWLS; Carrow-Woolfolk, 1996) 

❖ Receptive One-Word Picture Vocabulary Test (ROWPVT; Brownell, 

2000) 

❖ Woodcock-Johnson Picture Vocabulary Test (WJPV; Woodcock, 
McGrew, & Mather, 2001) 

Table ES.l: Data Collection Schedule 



Data Collection 


Respondent 


Summer 

2008 


Fall 

2008 


Spring 

2009 


Type of Data 


Teacher Data Form 
(2008) 


Teachers 


X 


X 




Teacher background 
information 


Teacher Data Form 
(2009) 


Teachers 






X 


Descriptive information 
about instructional 
materials used and Sam 
and Pat implementation 


Student Intake 
Form 


Site Staff on 
Behalf of 
Students 




X 


X 


Student background 
information 


Reading and 
English Language 
Pre-Tests 


Students 




X 


X 


Pre-test data 


Reading and 
English Language 
Post-Tests 


Students 




X 


X 


Outcomes data 


Daily Student 
Attendance Sheets 


Teachers 




X 


X 


Dosage/exposure to 
instruction 


Classroom 

Observations 


Evaluation Staff 




X 


X 


Descriptive information 
about instruction in both 
groups 
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The basic analytic strategy for assessing the impacts of Sam and Pat was to 
compare reading and English language outcomes for students who were randomly 
assigned to either the Sam and Pat or the control group, after controlling for 
student and teacher background characteristics (e.g., gender and ethnicity). The 
average outcome in the control group represents an estimate of the scores that 
would have been observed in the Sam and Pat group if they had not received the 
intervention; therefore, the difference in outcomes between the Sam and Pat and 
control groups provides an unbiased estimate of the impacts of Sam and Pat. 

The Adult ESL Literacy Intervention: Sam and Pat 

The Sam and Pat textbook (Hartel, Lowry, & Hendon, 2006) is described by the 
developers as a basal reader or textbook that tailors the methods and concepts of 
the Wilson and Orton-Gillingham reading systems developed for native speakers 
of English (Wilson & Schupack, 1997; Gillingham & Stillman, 1997) to meet the 
needs of adult ESL literacy level learners. 3 Sam and Pat was designed to 
incorporate the following components of the Wilson/Orton-Gillingham systems: 

❖ A focus on moving students systematically and sequentially from simple 
to complex skills and materials; 

❖ The use of multisensory approaches to segmenting and blending 
phonemes (e.g., sound tapping); 

❖ An emphasis on alphabetics/decoding, fluency, vocabulary, and reading 
comprehension; 

❖ The use of sound cards and controlled text (wordlists, sentences, stories) 
for practicing skills learned; and 

♦> Continual review (cumulative instruction) of letters, sounds, and words 
already learned. 

However, when writing Sam and Pat, the developers made variations on the base 
reading systems to make the text useful and relevant to the adult ESL literacy 
population for which the text was designed. Specifically, Sam and Pat differs 
from the base reading systems on four dimensions: 

❖ The sequence in which the sounds of English are taught; 

❖ The words chosen for phonics and vocabulary study; 

❖ The simplification of grammar structures presented; and 

❖ The added bridging of systematic reading instruction to ESL instruction. 



3 Although there is no available research on the effectiveness of Sam and Pat, the textbook and its 
accompanying training and technical support is based on these two reading systems (Wilson & 
Orton-Gillingham), which have shown promise in teaching struggling readers (Adams, 1991; 
Clark & Uhry, 1995; Kavenaugh, 1991; Torgesen et ah, 2006). 
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Building on the components of the earlier reading systems, Sam and Pat was 
therefore designed to (1) sequence the teaching of English sound and spelling 
patterns to ESL students by moving from a focus on simple to complex literacy 
skills and materials, (2) provide a controlled basal that follows this sequence of 
patterns, (3) use a simplified grammar, (4) embed a controlled vocabulary that is 
relevant to the lives of this population of students, and (5) include a collection of 
stories that are based on simplified themes from daily life. 

There are two volumes of Sam and Pat, and the Volume 1 literacy textbook was 
evaluated by this study. It is organized into a total of 22 multi-component lessons. 
The lessons follow what the developers consider to be an optimal sequence for 
introducing English phonics and high-frequency English sight words to 
non-native speakers of English. However, the sequence in which English vowels 
and consonant sounds are introduced has been modified from that usually used in 
approaches such as the Wilson and Orton-Gillingham reading systems. For 
example, like the Wilson System, Sam and Pat begins with the short-a sound, but 
short-a is followed several lessons later by short-u, rather than short-i. This 
modification was made to provide the maximum sound contrasts for the short 
vowel sounds that are notoriously challenging for English language learners to 
discriminate. 

Although the current study was a large-scale effectiveness study, we took 
measures intended to facilitate the implementation of Sam and Pat. The Sam and 
Pat developers provided the teachers assigned to the Sam and Pat group with 
training and technical assistance on implementing Sam and Pat. The training was 
developed specifically for the study, and included a 3-day training before the start 
of the fall 2008 term and a 2-hour refresher webinar before the start of the winter 
2009 term. The technical assistance provided to all Sam and Pat teachers included 
a site visit to observe and provide feedback early in the fall term, biweekly phone 
calls during the first 2 months of the fall term, and additional assistance as needed 
in response to phone calls and e-mails from teachers. The developers also 
provided 1 day of individualized assistance in person early in the winter term to 
teachers who appeared to be having difficulty implementing Sam and Pat. 

Summary of Study Findings 

Two-thirds of Sam and Pat Classes Observed Demonstrated Evidence of 
Implementing Sam and Pat as Intended 

About two-thirds (65 percent) of the Sam and Pat classes observed met the 
study’s instructional fidelity criteria regarding the use of Sam and Pat materials 
and engagement in reading instruction. More specifically, these teachers met the 
following criteria that were established in collaboration with the developers 
before the study began: 
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♦> Sam and Pat materials must be used for a minimum of 1 hour of 
instruction per class day; 

❖ Each class day must include at least 1 hour of instruction in reading 
development; and 

❖ Each class day, instruction should occur in at least three of the reading 
development instructional areas (e.g., phonics, fluency, reading 
comprehension). 

Because we did not observe all hours of instruction throughout the term, we 
cannot determine how many hours of Sam and Pat instruction were received by 
each student. However, students in the Sam and Pat group met for an average of 
79 hours total over the course of the term (not shown in tables). The Sam and Pat 
developers recommended that the text be implemented for a minimum of 60 hours 
per term. 

More Reading instruction Observed in Sam and Pat Classes, While More 
English Language Instruction Observed in Control Classes 

The Sam and Pat classrooms spent more time on reading development instruction 
than control classrooms (66 percent vs. 19 percent of observed time intervals, 
respectively), and the difference was statistically significant (Figure ES.l). 
Conversely, the control classrooms spent more time on English language 
acquisition instruction than did Sam and Pat classrooms (68 percent vs. 

27 percent of observed time intervals, respectively), and this difference was 
statistically significant. The control classrooms also spent more time on functional 
reading, writing and math instruction (content related to English language 
acquisition instruction) than Sam and Pat classrooms (18 percent vs. 5 percent of 
observed time intervals, respectively). 4 



4 We can only characterize implementation by reporting that (1) 65 percent of Sam and Pat classes 
met the study’s fidelity criteria, and (2) significantly more reading instruction was delivered in 
these classes, as compared to the control group classes. 
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Figure ES.l: Percent of Observed Instructional Intervals 
Spent in Key Instructional Areas, by Group 



100 r 



90 - 
80 - 



jj|y Sam and Pat mean 
| Control mean 




Reading English Language Functional Reading, 

Development Acquisition Writing, and Math 



* Indicates a difference that is significant at the 0.05 level, based on a 2-tailed t-test. 

Notes: N =980 observation intervals for Sam and Patgroup and 1,034 intervals for control group. Details 
may not sum to totals. Practices may be coded under multiple instructional areas during any one interval. 
Source: Adult ESL Literacy Impact Study classroom observation protocol. 



Students Made Gains, but There Were No Overall Impacts of Sam and Pat 
on Students’ Reading and English Language Skills 



On average, students participating in the study made statistically significant gains 
over the course of the term (effect sizes of 0.23 to 0.40). These gains are 
equivalent to 1 to 2 months of growth on the reading assessments, and 5 to 
6 months of growth on the English language assessments. 5 However, there were 
no statistically significant impacts of Sam and Pat on the reading and English 
language outcomes measured for the overall sample (Figure ES.2). Effect sizes 
ranged from -0.06 to 0.01. 



5 It should be noted that publisher guidelines for the grade and age equivalent calculations used to 
determine months of gains are based on norming populations that differ from the study population. 
(The WJ assessments were normed on a nationally representative sample of U.S. residents aged 
2 to 90+; the OWLS on a representative U.S. sample aged 3 to 21 years; and the ROWPVT on a 
representative U.S. sample aged 2 to 18 years.) No norming data exist for low-literate adult ESL 
learners. Additionally, the study used simplified or translated testing instructions when students 
did not appear to understand the tester’s directions. For these reasons, the number of months of 
growth should be interpreted with caution. 
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Figure ES.2: Impact of Sam and Pat on Reading and English 
Language Skills: Differences Between Sam and Pat and 
Control Groups at the End of the Term 

Reading 



Woodcock Johnson Letter Word 
Identification 


-0.030 A 
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Woodcock Johnson Word Attack 
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SARA Decoding 


-0.014 1 
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Comprehension 


-°.° 49 El 
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English Language 




— 


OWLS Listening 
Comprehension 
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ROWPVT 


- 0.065 


- 


Woodcock Johnson 
Picture Vocabulary ^ ^ 


J 1 1 


■ 0.012 

5 i i i i i i 



-0.30-0.25-0.20-0.15-0.10-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 
Impact (in Effect Size) 

Notes: N =580 for Sam and Pat group and 557 for control group. No impacts were statistically significant at 
the 0.05 level. 

Source: Adult ESL Literacy Impact Study student intake forms and assessments administered at the 
beginning and end of each term (fall 2008 and spring 2009), and fall 2008 teacher data form. 

No Impacts of Sam and Pat on Reading and English Language Outcomes 
Found for Subgroups Based upon Student Native Language and Cohort 

There were no statistically significant impacts found for students with a non- 
Roman-based alphabet background, native Spanish speakers, students from the 
first study cohort, or students from the second study cohort. Effect sizes ranged 
from -0.14 to 0.09. 

Some Suggestive Evidence of a Positive Impact on Reading Outcomes for 
Lower Literacy Students 

No statistically significant impacts were found for the students in the sample with 
relatively higher literacy levels (effect sizes ranged from -0.08 to 0.03). However, 
there was a suggestive finding for students who tested in the lower literacy score 
range at the beginning of the term. Within this subgroup, Sam and Pat group 
students scored higher on the Woodcock Johnson word attack (decoding) 
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assessment than control group students (effect size = 0.16). Because this 
difference was not statistically significant after adjusting for multiple 
comparisons, however, it is possible that the effect is due to chance alone. No 
impacts were found for the lower literacy students on the other reading and 
English language outcomes measured. 

Student Exposure to Reading or English Language Instruction Unrelated to 
Most Reading and English Language Outcomes Measured, Although Weak 
Relationships Found Between Exposure to Instruction and One English 
Language Outcome 

Student exposure to instruction was measured by the combination of reading and 
English language instruction provided in study classes and the number of hours 
students attended study classes. No statistically significant relationships were 
found between exposure to instruction and any of the reading outcomes measured 
and two of the three English language outcomes measured. However, the amount 
of exposure to English language instruction was positively and statistically 
significantly correlated with ROWPVT scores. The opposite pattern was found for 
reading instruction; exposure to reading instruction had a negative and statistically 
significant relationship with scores on the ROWPVT. However, the standardized 
coefficients in both cases were small (0.034 and -0.032, respectively). As an 
example, the 0.034 coefficient on the ROWPVT assessment indicates that, after 
controlling for total student attendance hours, an increase of 10 percent in the 
number of English language instruction intervals a student attended is associated 
with a 0.34 point increase on the test (which had a sample mean of 29). In 
addition, similar to the student attendance results, we cannot rule out the 
possibility that the statistically significant relationships were driven by other 
factors. Therefore, these findings should be interpreted with caution. 

Generalizability of the Study Findings 

The findings reported in this summary are limited to the specific intervention 
tested (Sam and Pat, v. 1) as implemented within the types of sites included in the 
study. For example, the study was implemented in sites large enough to offer at 
least 2 literacy level classes at the same time and location, within a subset of 
states that have the highest adult ESL enrollments. It is not known whether, or 
how, the results may generalize to other contexts. 
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